Skip to content

Add support for Scalar Quantization in HNSW.#673

Open
rahul-iyer wants to merge 2 commits into
LadybugDB:mainfrom
rahul-iyer:quantized-rerank-cache
Open

Add support for Scalar Quantization in HNSW.#673
rahul-iyer wants to merge 2 commits into
LadybugDB:mainfrom
rahul-iyer:quantized-rerank-cache

Conversation

@rahul-iyer

Copy link
Copy Markdown
Contributor

Summary

This PR adds scalar quantization support for HNSW indexes and improves the cache_embeddings = true query path so quantized embeddings use a dense in-memory cache with transaction-aware invalidation.

The main behavioral goal is to keep quantized search on the fast cached path while preserving correctness around inserts, rollback, commit, and MVCC visibility.

Benchmark

Ran it on OpenAI 50k small

Results:

Variant Load Index Build Ingest Query Latency DB Size Bytes/Vec Query RSS Recall@10 Precision@10
base 5.862s 51.000s 56.862s 17.812 ms/query 606.5 MiB 12718.7 746.6 MiB 0.9960 0.9960
sq8 5.765s 13.111s 18.875s 16.366 ms/query 723.4 MiB 15170.2 327.1 MiB 0.9460 0.9460
sq16 5.421s 23.337s 28.758s 16.444 ms/query 908.0 MiB 19041.5 489.4 MiB 0.9950 0.9950

Speedup vs baseline:

Variant Load Index Build Ingest Query Latency Query RSS Reduction
sq8 1.02x 3.89x 3.01x 1.09x 2.28x lower
sq16 1.08x 2.19x 1.98x 1.08x 1.53x lower

Storage overhead vs baseline:

Variant DB Size Bytes/Vec
sq8 1.19x 1.19x
sq16 1.50x 1.50x

Notes

  • sq8 gives the largest build-time win and the lowest query RSS, with lower recall due to 8-bit quantization.
  • sq16 keeps recall close to baseline while still improving index build, ingest, query latency, and query RSS.
  • The benchmark uses cache_embeddings = true, which is the intended optimized path for quantized HNSW search in this PR.

@rahul-iyer

Copy link
Copy Markdown
Contributor Author

Depends on LadybugDB/extensions#28

@rahul-iyer rahul-iyer changed the title Quantized rerank cache Add support for Scalar Quantization in HNSW. Jul 9, 2026
@adsharma

Copy link
Copy Markdown
Contributor

Some notes that may be helpful to people reviewing this code:

Vector extension (quantized HNSW)

Before this commit, at submodule 1bf007bca:

  • Quantized embeddings were NOT persisted to a dedicated disk table. QuantizedOnDiskEmbeddings read raw float embeddings from the node table and quantized them in memory on every access. There was no quantizedEmbeddingsTableID stored in HNSWStorageInfo.
  • The cache was a simple member variable (quantizedEmbeddingsCache / quantizedEmbeddingsCacheNumNodes) that was bluntly reset after every commitInsert() or finalize() call.
  • Checkpoint did not touch quantized data — there was no quantized embeddings table to checkpoint.
  • No concurrency handling — the cache was not shared across threads or queries in a safe way.

The new code at c0b7d47 adds:

  • Persistent quantized embeddings table: Quantized embeddings are now stored to a dedicated NodeTable (quantizedEmbeddingsTable), serialized in HNSWStorageInfo, and checkpointed. - Thread-safe shared cache: The QuantizedEmbeddingsCacheState uses a mutex and reference-counted shared pointer, allowing multiple concurrent queries to share the same cached column.
  • Transaction-aware invalidation: Cache is only invalidated on actual commit, not on every write.
  • Version-gated reads: The cache carries a version (commit timestamp) allowing readers to determine if the cache is up-to-date for their snapshot.
Hook When it fires Purpose Can write transactional data?
finalize After inserts, before commit (operator pipeline) Finish building index structures, write data ✅ Yes (undo buffer / WAL active)
commitCallback Inside publishCommit(), after WAL written Side effects on confirmed commit (cache invalidation, notifications) ❌ No (too late for WAL)
rollbackCallback Inside rollback() Side effects on confirmed rollback (clean up dirty state) ❌ No (transaction is aborting)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants